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SOFT (‘Second-Order Functions and Theorems’) is a tool to mimic second-order functions and the¬ 
orems in the first-order logic of ACL2. Second-order functions are mimicked by first-order functions 
that reference explicitly designated uninterpreted functions that mimic function variables. First-order 
theorems over these second-order functions mimic second-order theorems universally quantified over 
function variables. Instances of second-order functions and theorems are systematically generated 
by replacing function variables with functions. SOFT can be used to carry out program refinement 
inside ACL2, by constructing a sequence of increasingly stronger second-order predicates over one 
or more target functions: the sequence starts with a predicate that specifies requirements for the target 
functions, and ends with a predicate that provides executable definitions for the target functions. 


1 The SOFT Tool 

SOFT (‘Second-Order Functions and Theorems’) is a tool to mimic second-order functions and theo¬ 
rems 14] in the first-order logic of ACL2 Q. Second-order functions are mimicked by first-order func¬ 
tions that reference explicitly designated uninterpreted functions that mimic function variables. First- 
order theorems over these second-order functions mimic second-order theorems universally quantified 
over function variables. Instances of second-order functions and theorems are systematically generated 
by replacing function variables with functions. Theorem instances are proved automatically, via auto¬ 
matically generated functional instantiations |2]. 

SOFT does not extend the ACL2 logic. It is an ACL2 library, available in the ACL2 community 
books, that provides macros to introduce function variables, second-order functions, second-order theo¬ 
rems, and instances thereof. The macros modify the ACL2 state only by submitting sound and conserva¬ 
tive events; they cannot introduce unsoundness or inconsistency on their own. The main features of the 
macros are described and exemplified below; full details are in their documentation and implementation. 

1.1 Function Variables 

A function variable is introduced as 
(defunvar fv (* ... *) => *) 
where: 

• fv is a symbol, which names the function variable. 

• (*...*) is a list of 1 or more *s, which defines the arity, i.e. type Q, of fv. 

This generates the event 

(defstub fv (* ... *) => *) 

i.e. fv is introduced as an uninterpreted function with the given type. Furthermore, a table event is 
generated to record fv in a global table of function variables. 

For example. 
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(defunvar ?f (*) => *) 

(defunvar ?p (*) => *) 

(defunvar ?g (* *) => *) 

introduce two unary function variables and one binary function variable. Starting function variable names 
with ? provides a visual cue for their function variable status, but SOFT does not enforce this naming 
convention. 

1.2 Second-Order Functions 

SOFT supports three kinds of second-order functions: plain second-order functions, choice second-order 
functions, and quantifier second-order functions. 

1.2.1 Plain Functions 

A plain second-order function is introduced as 

(defun2 sof (/Uj ... fv^) (uj ... v,„) doc deal ... deal body) 
where: 

• so/ is a symbol, which names the second-order function. 

• ifv^ . . . fv^) is a non-empty list without duplicates of previously introduced function variables, 
whose order is immaterial, which are the function parameters of sof. 

• The other items are as in defun: individual variables, optional documentation string, optional 
declarations, and defining body. 

• f\/{body) U FV(measitT’e) U f\/{guard) = {/Uj,...,/u^}, where: 

- measure is fhe measure expression of sof, or nil if sof is nof recursive. 

- guard is fhe guard of sof (t if nof given explicifly in fhe declarafions). 

- FV( f erm) is fhe sef of funcfion variables fhaf eifher occur in term or are function parameters 
of second-order functions fhaf occur in term. 

I.e. fhe funcfion paramefers of sof are all and only fhe function variables fhaf sof depends on{^ 
This generafes fhe evenf 

(defun sof (vi . . . v,„) doc decl . . . decl body) 

i.e. sof is infroduced as a firsf-order funcfion using defun, removing fhe funcfion variables. Furfher- 
more, a table even! is generated fo record sof in a global fable of second-order funclions. 

For example, 

(defun2 quad[?f] (?f) (x) 

(?f (?f (?f (?f x))))) 

infroduces a non-recursive function fo apply ifs funcfion paramefer fo ifs individual paramefer four times. 
The name quad[?f] conveys fhe dependency on fhe funcfion paramefer and provides a visual cue for 
fhe implicif presence of fhe funcfion paramefer when fhe funcfion is applied, e.g. in (quad [?f ] x), buf 
SOFT does nof enforce fhis naming convenfion. 

As anofher example, 

'Thus, defun2 could have been defined to have the same form as defun, i.e. without (/U[ . . . fvf). However, the presence 
of the functions parameters parallels that of the individual parameters, and the redundancy check may detect user errors. 
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(defun2 all [?p] (?p) (1) 

(cond ((atom 1) (null 1)) 

(t (and (?p (car 1)) (all[?p] (cdr 1)))))) 

introduces a recursive predicate (i.e. boolean-valued function) that recognizes nil-terminated lists whose 
elements satisfy the predicate parameter. 

As a third example, 

(defun2 map[?f_?p] (?f ?p) (1) 

(declare (xargs :guard (all[?p] 1))) 

(cond ((endp 1) nil) 

(t (cons (?f (car 1)) (map[?f_?p] (cdr 1)))))) 

introduces a recursive function that homomorphically lifts ?f to operate on nil-terminated lists whose 
elements satisfy ?p. The predicate parameter ?p only appears in the guard, not in the body. 

As a fourth example, 

(defun2 fold[?f_?g] (?f ?g) (bt) 

(cond ((atom bt) (?f bt)) 

(t (?g (fold[?f_?g] (car bt)) (fold[?f_?g] (cdr bt)))))) 
introduces a generic folding function on values as binary trees. 

1.2.2 Choice Functions 

A choice second-order function is introduced as 

(defchoose2 sof (bv[ ... hvp) ifvi ... fVn) ("Ui ••• Vm) body key-opts) 
where: 

• so/ is a symbol, which names the second-order function. 

• (/uj . . . fvf) are the function parameters, as in def un2. 

• The other items are as in def choose: bound variables, individual variables, constraining body, 
and keyed options. 

• Vy{body) = {/ui,...,/u„}. 

This generates the event 

(defchoose sof (bvi ... bVp) (vi ... v^) body key-opts) 

i.e. sof is introduced as a first-order function using defchoose, removing the function variables. Fur¬ 
thermore, a table event is generated to record sof in the same global table where plain second-order 
functions are recorded. 

For example, 

(defchoose2 fixpoint[?f] x (?f) () 

(equal (?f x) x)) 

introduces a second-order function constrained to return a fixed poinf of ?f, if any exisfs. 
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1.2.3 Quantifier Functions 

A quantifier second-order function is introduced as 

(defun-sk2 sof (.fv^ ... fv^) (-ui ... Vyf) body key-opts) 
where: 

• so/ is a symbol, which names the second-order function. 

• (fv^ . . . fvf) are the function parameters, as in defun2 and def choose2. 

• The other items are as in def un-sk: individual variables, defining body, and keyed options. 

• FV(bocij/) U ^y^guard) = {/r/j,... ,/t/^}, where guard is the guard of sof (t if not given 
explicitly in the : witness-dels option). 

This generates the event 

(defun-sk sof (vi ... body key-opts) 

i.e. sof is introduced as a first-order function using defun-sk, removing the function variables. Fur¬ 
thermore, a table event is generated to record sof in the same global table where plain and choice 
second-order functions are recorded. 

For example, 

(defun-sk2 injective [?f] (?f) () 

(forall (x y) (implies (equal (?f x) (?f y)) (equal x y)))) 

introduces a predicate that recognizes injective functions. 

1.3 Instances of Second-Order Functions 

An instance of a second-order function is a function introduced as 

(defun-inst / (/Uj ... fvfi) (sof (fvf . ff) ... (fv^> . fj)) key-opts) 
where: 

• / is a symbol, which names the new function. 

• (/uj . . . fv^) are optional function parameters. If present, / is a second-order function; if absent, 
/ is a first-order function. 

• so/ is a previously introduced second-order function. 

• (’ • /i ’) • • • ’ • fm’ instantiation Z, i.e. an alist whose keys fv^ ’ are distinct 

function variables, whose values /, ’ are previously introduced function variables, second-order 
functions, or regular first-order functions, and where each f^ ’ has the same type as fv^ ’. Each fv^ ’ 
is a function parameter of so/. The notation (so/ (fv^’ ■ fi’) ... (fym’ ■ suggests 

the application of sof to the functions //; since the function parameters of sof are unordered, 
the application is by explicit association, not positional. An instance of a second-order function 
is introduced as a named application of the second-order function; SOFT does not support the 
application of a second-order function on the fly within a term, as in the application of a first-order 
function. Not all the function parameters of sof must be keys in Z; missing function parameters 
are left unchanged. 

• key-opts are keyed options, e.g. to override attributes of / that are otherwise derived from sof. 
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• If so/is a plain function, FV(r( body)) U f\/{L{measure)) U f\/{L{guard)) = {fv^,..., fv^}, 
where body, measure, and guard are the body, measure expression (nil if so/ is not recursive), 
and guard of so/, and L{term) is the result of applying £ to term (see below). 

• If so/ is a choice function, FV(r(body)) = {fv^,.. .,fv^}, where body is the body of so/. 

• If so/ is a quantifier function, FV(r(body)) U F\/{l.{guard)) = where body 

and guard are the body and guard of so/. 

This generates a defun, def choose, or defun-sk event, depending on whether so/ is a plain, choice, 
or quantifier function. The event introduces / with body r(body), measure l.{measure) (if so/ is 
recursive, hence plain), and guard L{guard) (if so/ is a plain or quantifier function). / is recursive iff 
so/ is recursive: defun-inst generates the termination proof of / from the termination proof of so/ 
using the techniques to instantiate second-order theorems described in Section [T3| 

Furthermore, defun-inst generates a table event to record / as the £ instance of so/ in a global 
table of instances of second-order functions. If / is second-order, defun-inst also generates a table 
event to record / in the global table of second-order functions. 

L{term) is obtained from term by replacing the keys of £ in term with their values in £. This 
involves not only explicit occurrences of such keys in term, but also implicit occurrences as func¬ 
tion parameters of second-order functions occurring in term. For example, if the pair (?f . f) is 
in £, sof [. . .?f. . .] is a second-order function whose function parameters include ?f, and term 
is (cons (?f x) (sof [. . .?f . . .] y)), then £(fer/?7) is (cons (f x) (sof [. . .f . . .] y)), where 
sof [. . . f . . . ] is the £’ instance of sof [. . . ?f . . . ], where £’ is the restriction of £ to the keys that are 
function parameters of sof [. . . ?f . . . ]. The table of instances of second-order functions is consulted 
to find sof [. . .f . . .]. If the instance is not in the table, defun-inst fails: the user must introduce 
sof [. . . f . . . ], via a defun-inst, and then re-try the failed instantiation. 

For example, given a function 

(defun wrap (x) (list x)) 
that wraps a value into a singleton list, 

(defun-inst quad[wrap] 

(quad[?f] (?f . wrap))) 

introduces a function that wraps a value four times. 

As another example, given a predicate 

(defun octetp (x) (and (natp x) (< x 256))) 
that recognizes octets, 

(defun-inst all[octetp] 

(all[?p] (?p . octetp))) 

introduces a predicate that recognizes nil-terminated lists of octets. 

As a third example, 

(defun-inst map[code-char] 

(map[?f_?p] (?f . code-char) (?p . octetp))) 

introduces a function that translates lists of octets to lists of corresponding characters. The replacement 
code-char of ?f induces the replacement octetp of ?p, because the guard of code-char is (equivalent 
to) octetp; the name map [code-char] indicates only the replacement of ?f explicitly. 

As a fourth example. 
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(defun-inst fold[nfix_plus] 

(fold[?f_?g] (?f . nfix) (?g . binary-+))) 

adds up all the natural numbers in a tree, coercing other values to 0. 

As a fifth example, given a function 

(defun twice (x) (* 2 (fix x))) 

that doubles a value, 

(defun-inst fixpoint[twice] 

(fixpoint[?f] (?f . twice))) 

introduces a function constrained to return the (only) fixed point 0 of twice. 

As a sixth example, 

(defun-inst injective[quad[?f]] (?f) 

(injective[?f] (?f . quad[?f]))) 

introduces a predicate that recognizes functions whose four-fold application is injective. 

1.4 Second-Order Theorems 

A second-order theorem is a theorem whose formula depends on function variables, which occur in the 
theorem or are function parameters of second-order functions that occur in the theorem. Since function 
variables are unconstrained, a second-order theorem is effectively universally quantified over the function 
variables that it depends on. It is introduced via standard events like def thmj^ 

For example, 

(defthm len-of-map[?f_?p] 

(equal (len (map[?f_?p] 1)) (len 1))) 

shows that the homomorphic lifting of ?f to lists of ?p values preserves the length of the list, for every 
function ?f and predicate ?p. 

As another example, 

(defthm injective[quad[?f]]-when-injactive[?f] 

(implies (injective[?f]) (injective[quad[?f]])) 

:hints 

(("Goal" :use 
((:instance 

injective[?f]-necc 

(x (?f (?f (?f (?f (mv-nth 0 (injective[quad[?f]]-witness))))))) 

(y (?f (?f (?f (?f (mv-nth 1 (injective[quad[?f]]-witness)))))))) 

(:instance 
injective[?f] -necc 

(x (?f (?f (?f (mv-nth 0 (injective[quad[?f]]-witness)))))) 

(y (?f (?f (?f (mv-nth 1 (injective [quad[?f]]-witness))))))) 

(:instance 
injective[?f] -necc 

^The absence of an explicit quantification over function variables in second-order theorems parallels the absence of an 
explicit quantification over individual variables in first-order theorems. 



A. Coglio 


23 


(x (?f (?f (mv-nth 0 (injective[quad[?f]]-witness))))) 

(y (?f (?f (mv-nth 1 (injective[quad[?f]]-witness)))))) 

(:instance 
injective [?f]-necc 

(x (?f (mv-nth 0 (injective[quad[?f]]-witness)))) 

(y (?f (mv-nth 1 (injective[quad[?f]]-witness))))) 

(:instance 
injective [?f]-necc 

(x (mv-nth 0 (injective[quad[?f]]-witness))) 

(y (mv-nth 1 (injective[quad[?f]]-witness)))))))) 

shows that the four-fold application of an injective function is injective. 

As a third example, given a function variable 

(defunvar ?io (* *) => *) 

for an abstract input/output relation, a predicate 

(defun-sk2 atom-io [?f_?io] (?f ?io) () 

(forall X (implies (atom x) (?io x (?f x)))) 

:rewrite :direct) 

that recognizes functions ?f that satisfy the input/output relation on atoms, and a predicate 

(defun-sk2 consp-io [?g_?io] (?g ?io) () 

(forall (x yl y2) 

(implies (and (consp x) (?io (car x) yl) (?io (cdr x) y2)) 

(?io X (?g yl y2)))) 

:rewrite :direct) 

that recognizes functions ?g that satisfy the input/output relation on cons pairs when the arguments are 
valid outputs for the car and cdr components, 

(defthm fold-io [?f_?g_?io] 

(implies (and (atom-io[?f_?io]) (consp-io[?g_?io])) 

(?io X (fold[?f_?g] x)))) 

shows that the generic folding function on binary trees satisfies the input/output relation when its function 
parameters satisfy the predicates just introduced. 

1.5 Instances of Second-Order Theorems 

An instance of a second-order theorem is a theorem introduced as 

(defthm-inst thm (sothm (/Uj . /j) ... (fv^ . /„)) :rule-classes ...) 
where: 

• thm is a symbol, which names the new theorem. 

• sothm is a previously introduced second-order theorem. 

• ■ fO • • • is an instantiation £, where each/u; is a function variable that sothm 

depends on. The notation (sothm (fv^ . /j) . . . (fv^ . /^)) is similar to defun-inst. 

• The keyed option : rule-classes . . . is as in defthm. 
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This generates the event 

(defthm thm T,{formula) : rule-classes ... : instructions proof) 
where: 

• formula is the formula of so thm. 

• proof consists of two commands for the ACL2 proof checker to prove thm using sothm. 

The first command of proof is 

(:use (:functional-instance sothm (/Uj /j) ... /„) more-pairs)) 

i.e. thm is proved using a functional instance of sothm. The pairs that define the functional instance 
include not only the pairs that form £ (in list notation instead of dotted notation), but also, in more-pairs 
above, all the pairs (so/ /) such that sof is a second-order function that occurs in sothm and / is 
its replacement in thm (i.e. / is the £’ instance of sof, where £’ is the restriction of £ to the function 
parameters of sof). These additional pairs are determined in the same way as when £ is applied to 
formula (see Section [O] ): thus, the result of (: functional-instance . . .) above is '^{formula), 
and the main goal of thm is readily proved. 

The use of the functional instance reduces the proof of thm to proving that, for each pair, the re¬ 
placing function satisfies all the constraints of the replaced function. Since function variables are uncon¬ 
strained, nothing needs to be proved for the (/u,- /;) pairs. For each (sof f) pair in more-pairs, it 
must be proved that / satisfies the constraints on sof. If sof references another second-order function 
sof ’ that depends on some fv^, a further pair (sof ’ f ’) goes into more-pairs, where / ’ is the 
appropriate instance of so/ ’, so that the constraints on sof to be proved are properly instantiated. This 
further pair generates further constraints to be proved. To properly instantiate these further constraints, 
another pair (sof ’ ’ f ’ ’) goes into more-pairs, if sof ’ ’ is a second-order function referenced by 
sof ’ that depends on some fv^, and / ’ ’ is the appropriate instance of sof ’ ’. Therefore, more-pairs 
includes all the pairs (sof f) such that so/ is a second-order function that is directly or indirectly 
referenced by sothm and that depends on some /u,, and / is the appropriate instance of sof. 

If so/ is a quantifier second-order function, it references a witness function so/^ introduced by 
defun-sk. The defun-sk that introduces the instance / of sof also introduces a witness function 
/^ that is effectively an instance of so/^, but is not recorded in the table of instances of second-order 
functions because so/^ and /^ are “internal”. The pair (so/^ /^) goes into more-pairs as well. 

For each pair (sof /) in more-pairs, the constraints of sof are: the definition of sof if sof is a 
plain function; the constraining axiom of sof if so/ is a choice function; the definition of sof and the 
rewrite rule of sof if sof is a quantifier function (the rewrite rule of sof is generated by defun-sk; 
its default name is so/-necc if the quantifier is universal, so/-suff if the quantifier is existential). 
Instantiating these constraints yields the corresponding definitions, constraining axioms, and rewrite 
rules of /, by the construction of the instance / of sof. 

The second command of proof is 

(:repeat (:then (:use facts) :prove)) 

where facts includes the names of all the / functions in more-pairs, which are also the names of their 
definitions and constraining axioms; facts also includes the names of the rewrite rules for quantifier 
functions. This command runs the prover on every proof subgoal, after augmenting each subgoal with 
all the facts in facts. This command has worked on all the examples tried so far, but a more honed 
approach could be investigated, should some future example fail; since the constraints are satisfied by 
construction, this is just an implementation issue. 

For example. 
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(defthm-inst len-of-map[code-char] 

(len-of-map[?f_?p] (?f . code-char) (?p . octetp))) 

shows that map [code-char] preserves the length of the list. 

As another example, given instances 

(defun-inst injective[quad[wrap]] (injective[quad[?f]] (?f . wrap))) 
(defun-inst injective [wrap] (injective[?f] (?f . wrap))) 

the theorem instance 

(defthm-inst injective[quad[wrap]]-when-injective[wrap] 

(injective[quad[?f]]-when-injective[?f] (?f . wrap))) 

shows that quad [wrap] is injective if wrap is. 

An example instance of f old-io [?f _?g_?io] is in Section]^ 

1.6 Summary of the Macros 

defunvar, defun2, def choose2, and defun-sk2 are wrappers of existing events that explicate func¬ 
tion variable dependencies and record additional information. They set the stage for defun-inst and 
defthm-inst. 

defun-inst provides the ability to concisely generate functions, and automatically prove their ter¬ 
mination if recursive, by specifying replacements of function variables. 

def thm-inst provides the ability to concisely generate and automatically prove theorems, by spec¬ 
ifying replacements of function variables. 

2 Use in Program Refinement 

In program refinement [91], a correct-by-construction implementation is derived from a requirements 
specification via a sequence of intermediate specifications. Shallow pop-refinement (where ‘pop’ stands 
for ‘predicates over programs’) is an approach to program refinement, carried out inside an interac¬ 
tive theorem prover by constructing a sequence of increasingly stronger predicates over one or more 
target functions. The sequence starts with a predicate that specifies requiremenfs for fhe fargef func¬ 
tions, and ends wifh a predicafe fhaf provides execufable definitions for fhe fargef funcfions. Shallow 
pop-refinemenf is a form of pop-refinemenf [8] in which fhe programs predicated upon are shallowly 
embedded functions of fhe logic of fhe fheorem prover, insfead of deeply embedded programs of a pro¬ 
gramming language as in Il8l. 

SOFT can be used fo carry ouf shallow pop-refinemenf in ACL2, as explained and exemplified below. 
The example derivafion is overkill for fhe simple program obfained, which can be easily wriffen and 
proved correcf direcfly. Buf fhe purpose of fhe example is fo illusfrafe techniques fhaf can be used 
fo derive more complex programs, and how SOFT supporfs fhese fechniques (which are more direcfly 
supporfed in higher-order logic). The hinfs in some of fhe fheorems below disfill fheir proofs info paffems 
fhaf should apply fo similarly strucfured derivafions, suggesfing opporfunifies for fufure automation. 

2.1 Specifications as Second-Order Predicates 

Requiremenfs over n> \ fargef funcfions are specified by infroducing funcfion variables fvi ,..., fv^ 
fhaf represenf fhe large! funcfions, and by defining a second-order predicafe specg over fv ^,..., fv^ fhaf 
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asserts the required properties of the target functions. The possible implementations are all the n-tuples 
of executable functions that satisfy the predicate. The task is to find such an n-tuple, thus constructively 
proving the predicate, existentially quantified over fhe funcfion paramefers. 

For example, given a funcfion 

(defun leaf (e bt) 

(cond ((atom bt) (equal e bt)) 

(t (or (leaf e (car bt)) (leaf e (cdr bt)))))) 
fo fesf whefher somefhing is a leaf of a binary free, a function fo exfracf from a binary free fhe leaves fhaf 
are nafural numbers, in no parficular order and possibly wifh duplicates, can be specified as 
(defunvar ?h (*) => *) 

(defun-sk io (x y) ; input/output relation 

(forall e (iff (member e y) (and (leaf e x) (natp e)))) 

:rewrite ;direct) 

(defun-sk 2 spec [?h] (?h) () 

(forall X (io x (?h x))) 

:rewrite :direct) 

The fask is to solve spec [?h] for ?h, i.e. to find an execufable funcfion h such fhaf fhe insfance spec [h] 
of spec [?h] holds. 

Properfies implied by fhe requiremenfs are proved as second-order fheorems wifh spec^ as hypofhe- 
sis, e.g. for validation purposes. Since fhe funcfion paramefers are universally quantified in fhe fheorem, 
fhe properfies hold for all fhe implemenfafions of fhe specification. 

For example, fhe members of fhe oufpuf of every implemenfafion of spec [?h] are nafural numbers: 

(defthm natp-of-member-of-output 

(implies (and (spec[?h]) (member e (?h x))) (natp e)) 

:hints (("Goal" :use (spec [?h]-necc (:instance io-necc (y (?h x))))))) 

2.2 Refinement as Second-Order Predicate Strengthening 

The specification specg is stepwise refined by consfrucfing a sequence specj,..., spec^^^ of increasingly 
sfronger predicafes over fv^ /u„. Each such predicate embodies a decision fhaf eifher narrows down 
the possible implementations or rephrases their description towards their determination. The correctness 
of each step 7 G {1,... ,m} is expressed by the second-order theorem (implies {specj) ispecj_i')'). 
The sequence ends with spec,^ asserting that each /u,- is equal to some executable function 
(defun-sk 2 defi ifv\) 0 (forall x (equal (/Uj x) (/^ x)))) 

(defun-sk 2 def„ ifv„) 0 (forall x (equal (fv^ x) (/„ x)))) 

(defun 2 spec^ (/Uj ... /u„) () (and (defi) ... (def„))) 

The tuple (/i, ■ • • ,/„) is the implementation. Chaining the implications of the m step correctness the¬ 
orems yields the second-order theorem (implies {spec^) (speco)). Its £ instance, where £ is the 
instantiation ((/uj . /j) ... (fv^ . /„)), is essentially £((specQ)) (because £((spec„,)) is trivially 
true), which asserts that the implementation (/j,..., /„) satisfies specg. 

More precisely, in fhe course of fhe derivation, funcfion variables may be added 

fo represenf additional largel funclions ,..., called by ,..., This may happen as fhe fask 

^The body of each (def un-sk2 def ,■ . . .) is a first-order expression of the second-order equality 
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of finding /i,.. • is progressively reduced to simpler sub-tasks of finding /„+i,. • ■,fn+p- If f'^n+k 
is added at refinement step j, since speCj_^ does not depend on the universal quantification of 

over the step correctness theorem (implies ispecj) (specy_j)) is equivalent to an existential 
quantification of over the hypothesis {specj) of the theorem. The complete implementation that 
results from the derivation is (/j,..., ,..., f^+p)- 

The function variables /k, are placeholders for the target functions in the specj predicates. Each 
fvi remains uninterpreted throughout the derivation; no constraints are attached to it via axioms. Each 
speCj is defined, so it does not introduce logical inconsistency. Inconsistent requirements on the target 
functions amount to spec^ being always false, not to logical inconsistency. Obtaining an implementation 
witnesses the consistency of the requirements. 

Eor example, spec [?h] from Section [2T] can be refined as follows. 


Step 1 Since the target function represented by ?h operates on binary trees, spec [?h] is strengthened 


by constraining ?h to be the folding function on binary trees from Section 1.2.1 


(defun-sk2 def-?h-fold[?f_?g] (?h ?f ?g) () 

(forall X (equal (?h x) (fold[?f_?g] x))) 

:rewrite :direct) 

(defun2 sped [?h_?f _?g] (?h ?f ?g) () 

(and (def-?h-fold[?f_?g]) (spec[?h]))) 

(defthm stepl (implies (sped[?h_?f_?g]) (spec[?h])) 
:hints (("Goal" :in-theory ’(sped [?h_?f_?g])))) 


The predicate sped [?h_?f _?g] adds to spec[?h] the conjunct def-?h-f old [?f _?g]. Thus, the 
task of finding a solution for ?h is reduced to the task of finding solutions for ?f and ?g: instantiating 
def-?h-f old [?f _?g] with solutions for ?f and ?g yields a solution for ?h, in Step 5 below. 


Step 2 The theorem f old-io [?f _?g_?io] from Section [E4} which shows the correctness of the 
folding function (with respect to an input/output relation) under suitable correctness assumptions on the 
function parameters, is instantiated with the input/output relation io used in spec [?h]: 


(defun-inst atom-io[?f] (?f) (atom-io[?f_?io] (?io . io))) 

(defun-inst consp-io [?g] (?g) (consp-io [?g_?io] (?io . io))) 

(defthm-inst fold-io [?f_?g] (fold-io [?f_?g_?io] (?io . io))) 

Since the conclusion (io x (fold[?f_?g] x)) of fold-io [?f_?g] equals the matrix (io x (?h 
x)) of spec [?h] when def-?h-f old [?f_?g] holds, sped [?h_?f _?g] is strengthened by replacing 
the spec [?h] conjunct with the hypotheses of f old-io [?f _?g]: 

(defun2 spec2[?h_?f_?g] (?h ?f ?g) () 

(and (def-?h-fold[?f_?g]) (atom-io [?f]) (consp-io [?g]))) 

(defthm step2 (implies (spec2 [?h_?f_?g]) (sped [?h_?f_?g])) 

:hints (("Goal" :in-theory ’(sped[?h_?f_?g] spec2 [?h_?f_?g] spec[?h] 

def-?h-fold [?f_?g]-necc fold-io[?f_?g])))) 


Step 3 The predicate atom-io [?f ] specifies requirements on ?f independently from ?g and ?h. An 
implementation f can be derived by constructing a sequence of increasingly stronger predicates over ?f, 
in the same way in which spec [?h] is being refined stepwise. This is a possible final result: 
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(defun f (x) (if (natp x) (list x) nil)) 

(defun-inst atom-io[f] (atom-io[?f] (?f . f))) 

(defthm atoni-io[f]! (atom-io [f ])) 

The predicate spec2 [?h_?f _?g] is strengthened by replacing the atom-io [?f ] conjunct with one that 
constrains ?f to be f: 

(defun-sk2 def-?f (?f) () (forall x (equal (?f x) (f x))) :rewrite :direct) 
(defun2 spec3[?h_?f_?g] (?h ?f ?g) () 

(and (def-?h-fold[?f_?g]) (def-?f) (consp-io[?g]))) 

(defthm step3-lemma (implies (def-?f) (atom-io[?f])) 

:hints (("Goal" :in-theory ’(atom-io[?f] atom-io[f]-necc 

atom-io [f]! def-?f-necc)))) 

(defthm step3 (implies (spec3[?h_?f_?g]) (spec2[?h_?f_?g])) 

:hints (("Goal" :in-theory ’(spec2[?h_?f_?g] spec3 [?h_?f_?g] step3-lemma)))) 

Step 4 The predicate consp-io [?g] specifies requirements on ?g independently from ?f and ?h. An 
implementation g can be derived by constructing a sequence of increasingly stronger predicates over ?g, 
in the same way in which spec [?h] is being refined sfepwise. This is a possible final resulf: 

(defun g (yl y2) (append yl y2)) 

(defun-inst consp-io [g] (consp-io [?g] (?g . g))) 

(defthm member-of-append ; used to prove CONSP-IO[G]-LEMMA below 
(iff (member e (append yl y2)) (or (member e yl) (member e y2)))) 

(defthm consp-io[g]-lemma ; used to prove CONSP-IO [G]! below 
(implies (and (consp x) (io (car x) yl) (io (cdr x) y2)) 

(io X (g yl y2))) 

:hints (("Goal" :in-theory (disable io) :expand (io x (append yl y2))))) 
(defthm consp-io[g]! (consp-io[g]) :hints (("Goal" :in-theory (disable g)))) 

The predicafe spec3 [?h_?f _?g] is sfrengfhened by replacing fhe consp-io [?f] conjuncf wifh one 
fhaf consfrains ?g fo be g: 

(defun-sk2 def-?g (?g) 0 

(forall (yl y2) (equal (?g yl y2) (g yl y2))) 

:rewrite :direct) 

(defun2 speed [?h_?f_?g] (?h ?f ?g) () 

(and (def-?h-fold[?f_?g]) (def-?f) (def-?g))) 

(defthm step4-lemma (implies (def-?g) (consp-io [?g])) 

:hints (("Goal" :in-theory ’(consp-io[?g] consp-io [g]-necc 

consp-io[g]! def-?g-necc)))) 

(defthm step4 (implies (speed [?h_?f_?g]) (spec3 [?h_?f_?g])) 

:hints (("Goal" :in-theory ’(spec3[?h_?f_?g] speed[?h_?f_?g] stepd-lemma)))) 

Step 5 Substituting the solutions f and g into fold [?f_?g] yields a solution for ?h: 

(defun-inst h (fold[?f_?g] (?f . f) (?g . g))) 

(defun-sk2 def-?h (?h) () (forall x (equal (?h x) (h x))) :rewrite :direct) 

The conjunct def-?h-f old [?f _?g] of speed [?h_?f_?g] is replaced with def-?h, which is equiva¬ 
lent to def-?h-f old [?f _?g] given the conjuncts def-?f and def-?g: 
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(defun2 spec5 [?h_?f_?g] (?h ?f ?g) () (and (def-?h) (def-?f) (def-?g))) 
(defthm stepS-lemma 

(implies (and (def-?f) (def-?g)) (equal (h x) (fold[?f_?g] x))) 

:hints (("Goal" :in-theory ’(h fold[?f_?g] def-?f-necc def-?g-necc)))) 
(defthm step5 (implies (spec5 [?h_?f_?g]) (speed [?h_?f_?g])) 

:hints (("Goal" :in-theory ’(speed[?h_?f_?g] spee5[?h_?f_?g] 

def-?h-fold [?f_?g] def-?h-neee step5-lemma)))) 

This concludes the derivation: spec [?h_?f _?g] provides executable solutions for ?h, ?f, and ?g. The 
resulting implementation is (h,f ,g). Chaining the implications of the step correctness theorems shows 
that these solutions satisfy the requirements specification: 

(defthm chain[?h_?f_?g] (implies (spec5[?h_?f_?g]) (spec[?h])) 

:hints (("Goal" :in-theory ’(stepl step2 step3 stepd step5)))) 

More explicitly, instantiating the end-to-end implication shows that h satisfies fhe requiremenfs specifi- 
cafion: 

(defun-inst def-h (def-?h (?h . h))) 

(defun-inst def-f (def-?f (?f . f))) 

(defun-inst def-g (def-?g (?g . g))) 

(defun-inst spec5[h_f_g] (spec5[?h_?f_?g] (?h . h) (?f . f) (?g . g))) 
(defun-inst spec[h] (spec[?h] (?h . h))) 

(defthm-inst chain[h_f_g] (chain[?h_?f_?g] (?h . h) (?f . f) (?g . g))) 
(defthm spec5 [h_f_g]! (spec5 [h_f_g]) 

:hints (("Goal" :in-theory ’(spec5 [h_f_g])))) 

(defthm spec[h]! (spec[h]) 

:hints (("Goal" :in-theory ’(chain[h_f_g] spec5[h_f_g]!)))) 

3 Related Work 

The instance-of-def spec fool llTdl and fhe make-generic-theory fool ifTTl aufomafically gener¬ 
ate insfances of functions and fheorems fhaf reference funefions consfrained via encapsulafion ifTSl . by 
replacing fhe consfrained funefions wifh funefions fhaf safisfy fhe consfrainfs. The insfanfiafion mecha¬ 
nisms of fhese fools are similar fo fhe ones of SOFT; consfrained funefions in fhese fools parallel function 
variables in SOFT. However, in SOFT function variables are unconsfrained; consfrainfs on fhem are ex¬ 
pressed via second-order predicates (fypically wifh quanfifiers), and fhe same function variables can 
be used as parameters of differenf consfraining predicates. Unlike SOFT, instance-of-def spec and 
make-generic-theory do nof handle choice and quantifier funefions, and do nof generate termination 
proofs for recursive funefion insfances. SOFT generafes one funefion or fheorem insfance af a fime, 
while instance-of-def spec and make-generic-theory can generate many. These fwo fools are 
more suited fo developing and insfanfiafing absfracf and parameterized fheories; SOFT is more suifed fo 
mimic second-order logic nofafion. 

The : consider hinf llT9ll heurisfically generafes funcfional insfanfiafions fo help prove given fhe¬ 
orems. SOFT generafes funefion and fheorem insfances for given replacemenfs of funefion variables; 
from fhese replacemenfs, fhe necessary funcfional insfanfiafions are generafed aufomafically. 

The def-functional-instance fool in fhe ACL2 communify books generafes fheorem insfances 
for given replacemenfs of funefions. This fool has more general use fhan SOFT’S defthm-inst, buf if 
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requires a complete functional instantiation, while def thm-inst only requires replacements for function 
variables. 

Wrapping existing events to record information for later use (as done by SOFT’S def unvar, def un2, 
def choose 2 , and defun-sk 2 ) has precedents. For example, the def :un-sk tool ifTTl is a wrapper of 
defun-sk that records information to help prove theorems involving quantifiers. It may be useful to 
combine def :un-sk with SOFT’S defun-sk2 wrapper. 

There are several tools to generate functions and theorems according to certain patterns, such as 
std: : def list in the ACL2 standard library and fty: : def list in the FTY library |[2^ . These tools 
may use SOFT to generate some of the functions and theorems as instances of pre-defined second-order 
functions and theorems. 

A general-purpose theorem prover like ACL2 can represent a variety of specification and refinement 
formalisms, e.g. KUminKIllISl [ 2011221 ; derivations can be carried out within the logic. But given the 
close ties to Applicative Common Lisp, a natural approach to program refinement in ACL2 is to specify 
requirements on one or more target ACL2 functions, and progressively strengthen the requirements until 
the functions are executable and performant. 

Alternatives to SOFT’S second-order predicates, for specifying requirements on ACL2 functions, in¬ 
clude encapsulate (possibly via the wrappers def spec and def abstraction in the ACL2 community 
books), def axiom, and def choose. But these are not as suited to program refinement: 

• An encapsulate involves exhibiting witnesses to the consistency of the requirements, which 
amounts to writing an implementation and proving it correct. But it is the purpose of program 
refinement to construct an implementation and its correctness proof. 

• A def axiom obviates witnesses but may introduce logical inconsistency. 

• A def choose obviates witnesses and is logically conservative, but: 

- It expresses requirements on single functions, necessitating the combination of multiple tar¬ 
get functions into one. 

- It expresses requirements on function results (the bound variables) with respect to function 
arguments (the free variables), but not requirements involving different results and different 
arguments, such as injectivity, non-interference lH^ . and other hyperproperties fT]. 

- It prescribes underspecified buf fixed function resulfs. For example, fhere is no clear refine- 
menf relation befween fhe function infroduced as (def choose f (y) (x) (>yx)) and fhe 
function infroduced as (defun g (x) (+ x 1)). 

In confrasf, a second-order predicafe can specify any kind of requiremenfs, on multiple funclions, main- 
faining logical consisfency, and doing so wifhouf premafure wifnesses. 

In fhe derivafion in Section [T2l fhe use and insfanfiafion of fhe generic folding funcfion on binary 
frees is an example of fhe applicafion of algorifhm schemas in program refinemenf, as in ll 2 n buf here 
realized via second-order funclions and Iheorems. Second-order funclions express algorifhm schemas, 
and second-order Iheorems show fheir correcfness under suilable condifions on fhe funcfion paramelers. 
Applying a schema adds a consfrainf fhaf defines a largel funcfion fo use fhe schema, and infroduces 
simpler largel funclions corresponding fo fhe funcfion paramelers, conslrained lo safisfy fhe condifions 
for fhe correcfness of fhe schema. 

A refinemenf slep from a specification specj can be performed manually, by wriling down specj_^_i 
and proving (implies ispeCj_^_^) {specj)). If is sometimes possible lo generafe specy^j from specy, 
along wifh a proof of (implies (specy_|_j) (specy)), using aulomafed fransformalion fechniques. Au¬ 
tomated Iransformalions may require parameters lo be provided and applicabilily conditions fo be proved. 
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but should generally save effort and make derivations more robust against changes in requirements spec¬ 
ifications. At Kestrel Institute, we are developing ACL2 libraries of automated transformations for pro¬ 
gram refinement. 


4 Future Work 


Guards defun-inst could be extended with the option to override the default guard L{guard) with a 
different guard’, generating the proof obligation (implies guard’ L{guard)). This would be useful 
in at least two situations. 

A first situation is when the function instance has more guard conditions to verify than the second- 
order function being instantiated, due to the replacement of a function parameter (which has no guards) 
with a function that has guards. Providing a stronger guard to the function instance would enable the 
verification of the additional guard conditions. For example, an instance quad [cdr] of quad [?f ] from 


Section 1.2.1 could be supplied with the guard (true-listp x). 

A second situation is when the guard of the second-order function being instantiated includes condi¬ 
tions on function parameters that involve a quantifier, e.g. fhe condifion fhaf fhe binary operafion ?op of 
a generic folding funcfion over lisfs is closed over fhe fype ?p of fhe lisf elemenfs. Insfanfiafing ?p wifh 
natp and ?op wifh binary-+ safisfies fhe condifion, buf L{guard) still includes a quantifier fhaf makes 
fhe insfance of fhe folding funcfion non-execufable. Supplying a guard’ fhaf rephrases L{guard) fo 
omif fhe satisfied closure condifion would solve fhe problem. As guard obligafions on individual param¬ 
eters are relieved when functions are applied fo ferms in a term, if makes sense fo relieve guard obligafions 
on funcfion parameters when second-order functions are “applied” fo functions in defun-inst. 

defun-inst could also be extended wifh fhe abilify fo use fhe insfances of fhe verified guard condi¬ 
tions of fhe second-order funcfion being instanfiafed, fo help verify fhe guard conditions of fhe funcfion 
insfance. This may completely verify fhe guards of fhe instance, when no guard overriding is needed. 


Partial Functions SOFT could be extended with a macro def pun2 to introduce partial second-order 
functions, mimicked by partial first-order functions introduced via defpun ifT^ . defun-inst could 
be extended to generate not only partial function instances, but also total function instances when the 
instantiated : domain or : gdomain restrictions are theorems. Partial second-order functions would be 
useful, in particular, to define recursive algorifhm schemas whose measures and whose argumenf updafes 
in recursive calls are, or depend on, funcfion paramefers. An example is a general divide-and-conquer 
schema0 


Mutual Recursion SOFT could be extended wifh a macro mutual-recursion2 fo infroduce mufu- 
ally recursive plain second-order funcfions (wifh defun2), mimicked by mufually recursive firsf-order 
functions infroduced via mutual-recursion, defun-inst could be extended to generate instances of 
mutually recursive second-order functions. 


Lambda Expressions defun-inst and defthm-inst could be extended to accept instantiations that 
map function variables to lambda expressions, similarly to : functional-instance. 


The folding function from Section 1.2.1 is a divide-and-conquer schema specialized to binary trees. 
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Instantiation Transitivity If sof ’ is introduced as the £ instance of sof, and f is introduced as the 
£’ instance of so/ \ then / should be the £” instance of sof, where £” is a suitably defined composition 
of £ and £’. Currently defun-inst does not record / as an instance of sof when f is introduced, but 
it could be extended to do so. With this extension, inj ective [quad [wrap] ] in Section 1.5 would be 
the ((?f . quad [wrap])) instance of injective [?f] in Section 1.2.3| 

In a related but different situation, given sof, sof ’, f, £, £’, and £” as above, but with / introduced 
as the £” instance of sof, and sof ’ introduced as the £ instance of sof, in either order (i.e. / then 
sof ’, or sof ’ then /), then / should be the £’ instance of sof ’. Currently defun-inst does not 
record / as an instance of sof ’ when / (after sof ’) or sof ’ (after /) is introduced, but could be 
extended to do so. With this extension, if injective [quad [wrap] ] were introduced as the ((?f . 
quad [wrap])) instance of inj ective [?f ], and injective [quad [?f ] ] were introduced as the ((?f 
. quad[?f])) instance of injective [?f] as in Section [L3| then injective [quad[wrap] ] would 
be the ((?f . wrap)) instance of injective [quad[?f]]. 

An alternative to these two extensions of defun-inst is to extend SOFT with a macro to claim 
that an existing instance of a second-order function is also an instance of another second-order function 
according to a given instantiation. The macro would check the claim (by applying the instantiation and 
comparing the result with the function) and extend the table of instances of second-order functions if the 
check succeeds. In the first scenario above, the macro would be used to claim that / is the £” instance of 
sof; in the second scenario above, the macro would be used to claim that / is the £’ instance of sof ’. 


Function Variable Constraints Currently the only constraints on function variables are their types, 
defunvar could be extended to accept richer signatures for function variables, with multiple-value re¬ 
sults and single-threaded arguments and results, defun-inst and def thm-inst would then be extended 
to check that instantiations satisfy these additional constraints. A more radical extension would be to at¬ 
tach logical constraints to certain function variables, as in encapsulations. 


Automatic Instances As explained in Section 1.3 when an instantiation is applied to a term, the 
table of instances of second-order functions is consulted to find replacemenfs for certain second-order 
functions, and the application of the instantiation fails if replacements are not found. Thus, all the needed 
instances must be introduced before applying the instantiation, e.g. in Section [T3] the two defun-insts 
had to be supplied before the last def thm-inst. SOFT could be extended to generate automatically the 
needed instances of second-order functions. 

SOFT could also be extended with a macro defthm2 to prove a second-order theorem via defthm 
and to record the theorem in a table, along with information about the involved second-order functions, 
defun-inst could be extended with the option to generate instances of the second-order theorems that 
involve the second-order function being instantiated. defthm2 could include the option to generate 
instances of the theorem that correspond to the known instances of the second-order functions that the 
theorem involves. These extensions would reduce the use of explicit def thm-insts. 

The convention of including function variables in square brackets in the names of second-order func¬ 
tions and theorems, could be exploited to name the automatically generated function and theorem in¬ 
stances, as suggested by the examples throughout the paper. 


Other Events SOFT could be extended to provide second-order counterparts of other function and 
theorem introduction events, e.g. define, defines, and defrule in the ACL2 community books. 
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